Segmenting memory colours

Memory colours refer to the colour of specific image classes that have the essential attribute of being perceived in a consistent manner by human observers. In colour correction -or renderingtasks, this consistency implies that they have to be faithfully reproduced; their importance, in that respect, is greater than other regions in an image. Before these regions can be properly addressed, one must in general detect them. There are various schemes and attributes to do so, but the preferred method remains to segment the images into meaningful regions, a task for which many algorithms exist. Memory colours’ regions are not, however, similar in their attributes. Significant variations in shape, size, and texture do exist. As such, it is unclear whether a single algorithm is the most adapted for all of these classes. In this work, we concern ourselves with three memory colours: blue sky, green vegetation, and skin tones. Using a large database of real-world images, we (randomly) select and manually segment 900 images that contain one of the three memory colours. The same images are then automatically segmented with four classical algorithms. Using class-specific Eigenregions, we are able to provide insights into the underlying structures of the considered classes and class-specific features that can be used to improve classification’s accuracy. Finally, we propose a distance measure that effectively results in determining how well is an algorithm is adapted to segment a given class. Introduction All image regions are not created equal; indeed, for digital photography some classes have a much greater importance than others. Some of the most important classes are the so-called memory colours: blue sky, green vegetation and skin tones [10]. It has been shown that human observers locate these classes in very specific areas of the colour gamut [2, 20]. Thus, many colour rendering and correction algorithms specifically try to map these colours to the correct values, such that the resulting images can be rendered. As a result, detecting these regions has been (and still is) a very active area of research. Detection algorithms generally rely on many different features to classify memory colours: approaches include the use of shape, size, position, colour, and texture [4, 21, 13, 3]. Prior to being detected, however, images have to be segmented into meaningful regions. How meaningful a region is depends on the intended application of the segmentation, but most segmentation evaluation methods are predicated on the idea(l) that all regions are of equal importance. As such, it is in their entirety that the resulting segmentations are compared to manually segmented images [19]. This work addresses the problem of class-specific segmentation evaluation, as well as the localisation of memory colour regions within natural images. Our framework builds on the eigenregions proposed by Fredembach et al. [11], which are principal component analysis (PCA) based geometrical features that encompass shape, size, and position information of regions. The central idea is to calculate class-specific eigenregions, i.e., obtaining different geometrical descriptors for each class. To be used in such a manner, the considered classes have to be reasonably localised across images, i.e., they should usually be found in similar position within images. The classes we consider here: blue sky, green vegetation, and skin tones generally fulfil, due to physics or photographic composition, this localisation criterion. An objective ground truth for our experiments is obtained by manually segmenting 900 images, 300 per class. These accurate binary segmentation maps are used to calculate class-specific eigenregions that are subsequently compared to the ones resulting from automatic segmentation of the same images. Four segmentation algorithms that employ very different information are compared: Meanshift (density estimation process) [5], Felzenswalb and Huttenlocher (minimum spanning trees) [8], k-means (Euclidian distances between clusters) [1], and edgeflow (Gabor filter banks) [14]. The comparison is based on the idea that if human segmentation is available for a given class, then its N eigenregions provide a reference basis in N-dimension. An algorithm-based segmentation of the same data will, however, provide a different basis in N-dimension. Measuring the distance between these bases effectively quantifies the performance of the algorithm. The results show a strong class-dependency in both the accuracy of segmentation and shape of the eigenregions. In fact, the proposed framework can be used to quantify, for a given class, the distance between any two algorithms and the influence of the algorithm’s parameters. In addition, it yields class-specific features that can be used for classification tasks. Background Segmentation-wise assessment of class-specific data is scarce. In a more global setting, however, assessing the performance of automatic segmentation is not a new concern and several approaches have been presented that yield a measure of “closeness” or “agreement” with human segmentation. Martin et al. [17] first proposed the use of region consistency over a database of human-segmented images [16] to evaluate the performance of automatic segmentation algorithms. These measures of segmentation consistency turned out to be biased toward overor under-segmentation, so in [15] the use of precision and recall on region boundaries was suggested instead. A benchmark of several segmentation algorithms based on precision and recall was published in [6]. A different, region-based consistency measure was presented by Ge et al. in [12]. Their measure also depends on the overlap between automatic and human segmentations, but it was computed on images that contained only two regions: a salient object and its background. Overlap was measured after deciding (based on the human segmentation) which subset of regions in the automatic segmentation best matched any given human region. More recently, Unnikrishnan et al. [19] presented a benchmark based on the Normalized Probabilistic Rand index. This measure compares segmentations through a soft weighting of pixel pairs that depends on the variability of the ground truth data. Other measures of segmentation consistency have been proposed in [9], [18], and [7]. A concise survey of these measures is provided in [19]. Despite their potential usefulness, each of the above methods for evaluation has its own limitations. First of all, they are global methods designed for entire image agreement, we are here concerned about specific classes. Boundary based methods will give good scores to under-segmented images in which two or more distinct (and possibly large) image regions are connected through narrow “leaks”. Since most of the boundary is recovered, boundary matching may falsely indicate that the segmentation is accurate. Methods based on overlap such as Ge et al. can be biased toward high scores by over-segmenting. In addition, this method assumes some form of expert is available to decide which of the over-segmented regions should be merged together to match human segmentation. The benchmark by Unnikrishnan et al. [19] provides interesting insights about the performance of segmentation methods on natural images, however, the question remains of whether particular algorithms are better for specific segmentation tasks, which is one of the fundamental problems addressed in this paper.